1. Field of the Invention
The present invention relates to an image pickup apparatus, an image processing apparatus, and an image processing method that perform comparison light composition of image data of a plurality of frames picked up in time series.
2. Description of the Related Art
In recent years, there has been proposed an image pickup apparatus such as a digital camera implemented with a comparison light composition processing function. The comparison light composition processing is processing for comparing pixel values in the same pixel position in images of a plurality of frames picked up in time series and replacing a pixel value in the pixel position with a pixel value having a larger value.
The comparison light composition processing is used for images of a plurality of frames obtained by picking up images in time series (so-called continuous photographing) at necessary and sufficient short fixed shutter speed (therefore, the shutter speed is common to the respective images) in a photographing scene including a relatively light and small moving body. The comparison light composition processing can acquire a track of the moving object as a satisfactory image having a high SN ratio.
Several examples of photographing scenes to which the comparison light composition processing is applied include fireworks, celestial photographing, and photographing of a moving track of a headlight of a car or the like at night.
For example, in the case of the firework photographing, images of flows in several seconds of light spots forming fireworks are picked up in time series at shutter speed shorter than the several seconds to acquire images of a plurality of frames. In the case of the celestial photographing, images of extremely slow movements of stars forming a celestial body are picked up in time series at short shutter speed to acquire images of a plurality of frames. Further, in the case of the moving track of the headlight of the car or the like at night, similarly, images of the moving track are picked up in time series at shutter speed shorter than shutter speed corresponding to the length of the moving track to be photographed to obtain images of a plurality of frames.
In all the cases, by performing comparison light composition of the obtained images of the plurality of frames, it is possible to avoid image quality deterioration due to a dark current as in the case of photographing at shutter speed for a long time and obtain an image having a high SN.
Incidentally, image noise is classified into fixed pattern noise attributable to a characteristic of each of pixels and random noise not having a correlation with a pixel position. The random noise is roughly classified into dark random noise that occurs irrespectively of a signal generated by photoelectric conversion according to incident light in a pixel (therefore, occurs even in a light-blocked state) and light random noise that occurs according to the signal generated by the photoelectric conversion (therefore, does not occur in the light-blocked state).
A magnitude relation of the dark random noise and the light random noise depends on an electric gain through exposure control of a camera. More specifically, when an amplification amount by the electric gain is small (when an exposure amount is large and there is little necessity of amplification), the light random noise increases and becomes predominant because a pixel value obtained by photoelectrical conversion in the respective pixels is relatively large. When the amplification amount is large (when the exposure amount is small and there is necessity of amplification), the light random noise decreases and the dark random noise becomes predominant because the pixel value obtained by the photographic conversion in the respective pixels is relatively small.
The dark random noise occurs, for example, as noise attributable to a dark current of a photodiode of a pixel or as circuit noise. The dark random noise is not only included in a pixel value obtained from an effective pixel but also included in an OB component obtained from an optical black (OB) region optically shielded from light on an image pickup surface of an image pickup device. The OB component neither includes a signal generated by photoelectrically converting incident light nor includes the light random noise. Therefore, it is possible to more easily and more accurately detect the dark random noise when the OB component is used than when the pixel value obtained from the effective pixel is used.
A value of the dark random noise fluctuates for each of frames at random. When the comparison light composition processing is performed, one comparison light image adopting a largest pixel value in a plurality of frames obtained by photographing is generated. However, since a pixel value including largest random noise is highly likely to be a largest pixel value, as the number of frames is larger, the pixel value is replaced with a larger value affected by larger random noise.
On the other hand, if the comparison light composition processing is performed on the OB component obtained from the OB region in the same manner as for the effective pixel region, a pixel value having magnitude corresponding to the magnitude of the number of frames is adopted. Therefore, by subtracting the OB component subjected to the comparison light composition processing from a comparison light image of the effective pixel region, it is possible to effectively cancel the influence of the random noise.
For example, Japanese Patent Application Laid-Open Publication No. 2013-240029 describes a technique for, in digital clamp processing for removing an offset component from a composed image, controlling a digital clamp amount on the basis of the number of composed photographed images to cope with fluctuation in a black level corresponding to the number of composed images in image composition such as comparison light composition and comparison dark composition by a RAW image and suppress image quality deterioration of the composed image. The publication further mentions that the digital clamp amount is controlled according to at least one of photographing sensitivity, a white balance, an exposure time, a correction amount of a peripheral light amount, and an environmental temperature.